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ABSTRACT 



This paper describes efforts at East Carolina University 
(North Carolina) to develop a survey instrument and related procedures to 
assess distance learning courses. The survey was conducted during the 1998-99 
academic year via the World Wide Web and combined the Student Opinion of 
Instruction Survey (SOIS) with additional questions that assessed the extent 
to which graduate -level distance education courses met Quality Assurance 
Standards for Distance Learning. This initial survey achieved a response rate 
of only 20 percent due to such factors as student's poor knowledge of the 
university's e-mail accounts, inadequate university help facilities, and time 
demands on respondents. However, the survey instrument itself proved 
successful in that all respondents answered all of the questions and none had 
negative comments about it. Modification of the survey and its 
re-administration in spring 1999 resulted in a 43 percent response rate, 
extensive comments, and general student satisfaction with distance education 
courses and the survey instrument. Findings highlight the difficulty of 
reaching distance education students with a Web-based survey, the complexity 
of developing an appropriate survey, and the need to address campus political 
problems in developing Web-based surveys. (DB) 
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Assessing Distance Learning Using a Website Survey 



This paper describes the efforts at East Carolina University to develop a survey instrument and 
related procedures to assess distance learning courses. The survey was conducted during the 
1998-99 academic year via the World Wide Web and combined the Student Opinion of 
Instruction Survey with questions developed by the ECU Graduate School that assessed the 
extent that graduate-level courses met Quality Assurance Standards for Distance Learning. The 
standards apply to graduate courses with a distance learning component that comprise 25% or 
more of the total allocated hours of the course. 

This paper describes: (1) the need to develop assessment tools for distance learning courses and 
provide other background information; (2) the ECU assessment instrument; (3) the 
implementation procedures using the World Wide Web; and (4) the survey results; and provides 
conclusions about the processes used. 




Assessing Distnnee Lenming Using n ^Vebsite Survey 



I. The need to develop assessment fo o ls for distance learning 
The rapid expansion of distance education in recent years has increased eoneems that quality 
standards and assessment techniques are developed. Some of these concerns arise through the 
accreditation process. A recent report prepared for the council for Higher Education 



Accreditation provided the following policy agenda for the accreditation community regarding 
distance education;* 



Establish reliable and valid performance measurements for distance learning; 

Inquire providers to substantiate evidence of contact between faculty and students • 

Require evidence of effective instructional techniques; 

Promote systematic efforts for selecting and training faculty; 

Assure the availability of learning resources; 



f'omoK ongoing minhoring and enhancement of, he iechnohgyinJhastn«:ture of imtimkms: 

Focus attention on the development of courseware and the availability of information- and 
Examine alternatives to the traditional accreditation process. 



East Carolina University, like other institutions, is offering a growing number of courses that are 
taught using one or more distance learning techniques antemet. Interactive video. Recorded 
video, etc.). Most of these have been graduate courses, although a growing number of 



undergraduate courses are also being offered with distance learning techniques. 



The ECU Graduate School has been very concerned about establishing procedures to 
ensure that courses offered via distance learning techniques maintain high quality comparable to 
courses offered in a traditional classroom setting. (Distance learning courses were defined as 



courses in which the distance learning component comprises 25 % or more of the total allocated 
contact hours.) A set of standards for distance learning classes was developed and each semester 
department chairs are required to make a series of assurances to the Graduate School before 



'Assuring Quality in Distance Learning, A Preliminary Review, a report prepared for the Council for Higher 
Education Accreditation by the Institute for Higher Education Policy, Washington, DC, April 1998. 



offering graduate level distance learning courses. The assurances require that: one faculty 
member be designated as the “course director;” the prerequisites clearly describe the hardware, 
software, and technical requirements for the students; adequate library resources be available; 
and the course be designed so that significant faculty-student and student-student interaction 
occur within the course. The department chair agrees that the “Quality Assurance Standards for 
Distance Learning” will be followed including providing an opportunity to the students for end 
of course assessment. 

A set of questions was developed to facilitate student assessment. It was envisioned that 
these questions would evolve into a standardized assessment tool that would be used by students 
at the end of the course. 

II. The ECU Assessment Instrument 

At the time that the Graduate School was developing an assessment tool, other groups on campus 
(including the Office of Planning and Institutional Research) were expressing concern that the 
Faculty Senate approved Student Opinion Of Instruction Survey (SOIS) was not completely 
appropriate for the growing number of distance learning courses (both graduate and 
vmdergraduate) and, therefore, not adequately meeting accreditation criteria. Not only were some 
of the questions inappropriate, but the data collection method (paper forms delivered, completed, 
and collected in the classroom) was clearly not appropriate for distance learning courses where 
the students are never assembled at a single time and place. The lack of an appropriate 
instrument meant that a growing number of distance learning courses were not surveyed by 
SOIS. Therefore, a number of faculty members were not receiving these data for use in the 
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personnel decisions affecting them and that the university was not able to assess effectively the 
quality of instruction of these courses in comparison to more traditional classroom courses. 

As a result of these concerns, a campus-wide group was formed to develop a new survey 
instrument for distance learning courses. This group decided to work together with the Graduate 
School to develop a single survey instrument that incorporated both Quality Assurance Standards 
and SOIS questions. Students of graduate distance learning courses would be asked to complete 
a single end of the course questionnaire, following which the results would be compiled into two 

reports- a quality assurance report to the Graduate School, and the standard SOIS report for the 
instructor. 

The resulting questionnaire was 48 questions long and there was concern about the length 
of the survey. However, in the University of North Carolina system the Sophomore and 
Graduating Senior surveys are much longer, but have good response rates. Partially because of 
its length, the survey was divided into six sections: Self Assessment, Instructor Assessment, 
Course Assessment, Distance Learning Assessment, Overall Course Assessment, and Written 
Comments. The questionnaire was designed to resemble the appearance of the paper SOIS form. 
Most of the questions use radio buttons for the responses and questions with the same response 
scale are grouped into tables. Two of the questions had wordy response choices so we used 
drop-down lists of response choices. 

The response scale was repeated frequently so that it was always visible on the screen. 

The last three questions are for written comments. Three boxes were provided for written 
comments. The box provided for the written comment will expand as needed. All of the SOIS 
questions approved for use in personnel decisions were included, although the wording of a 
number of the questions had to be modified slightly to make it appropriate for distance learning. 



III. Implementation Procedures Using the World Wide Web 

Traditional methods of administering the SOIS (administering a paper form during the last full 
week of classes) are inappropriate for distance learning courses. Sometimes the instructor 
interacts with the students only via the Internet or other distance learning medium and does not 
have an opportunity to bring the students together at one time and place to administer a survey. 
Given this situation conducting a survey via the Web has important advantages: 

• Paper forms do not have to be produced, pre-printed with course information, and 
delivered to the students. 

• Results do not have to be scanned. They can be stored directly in a database. 

• Responses can be checked for completeness or consistency as the survey is being 
completed. 

• Students can be given the opportunity to record extensive comments. 

There are also disadvantages: 

• It is nearly impossible to maintain both complete anonymity of results and to ensure 
that only one survey per each student entitled to participate in the survey. 

• There may be some bias introduced by favoring the computer-literate students. 

• The survey cannot be administered at a single time and place as with traditional 
survey techniques. 

Decisions had to be made on four critical issues early in the survey design. 

(1) How to communicate effectively with the distance learning students in requesting 
their participation in the survey? 

(2) How to provide anonymity of response? 



(3) How to control access to the survey to ensure that only those students who are 
registered for the course are able to complete the survey and that each student submits 
only one survey? 

(4) Where will the survey instrument reside? 

Communication with survey population . We chose two methods: (a) Send email 
messages to the student's ECU email account; and (b) Provide messages to instructors that the 
instructors would post to the course bulletin board or otherwise deliver to the student. 

Anonymity of response is an important characteristic of our standard SOIS. Ideally, we 
would like to preserve this characteristic in the web-based survey. It was recognized that 
complete anonymity would be difficult to achieve because of feasibility of tracing the computer 
from which the survey was submitted. Alternative anonymous identifiers were discussed, 
including the issuing of random numbers to students by the instructors. Therefore, our office 
would only have to know which numbers were issued. This idea was rejected this because of the 
amount of faculty involvement required at a time when faculty are already extremely busy. For 
this pilot survey, we decided to forego anonymity and simply pledge to the students that (a) their 
responses would be stored anonymously in a database, (b) the identifiers would only be used for 
survey control, and (c) survey results would be reported in aggregate form only. 

Choice of identifiers for survey control . This was one of the most difficult and important 
decisions made. Since we decided to forego anonymity, we needed to devise a method of 
ensuring that only eligible students could participate in the survey and that they can do so only 
once per course. Our choice of identifier was a combination of the student's email ID and their 
Personal Identification Number (PIN) that was issued to them. We preferred that to the Social 
Security Number (SSN) because of security concerns about passing SSN's over the Internet, 



and, more importantly, because it may be less threatening to the student. The PINs were 
anonymous to our office, although we could identify the student through his/her email ID. 

Location of the survey instrument . This issue is partially determined by the answer to the 
anonymity question. If the codes used for survey control are completely anonymous, then the 
survey can reside on any computer. If, however, you collect easily identifiable information as we 
did with the student's email ID, then the results should be stored on a secure server to ensure 
confidentiality. For that reason, we chose a secure server maintained by our computer center 
rather than our own server where we would have had a greater degree of control. 

Procedures followed: 

(1) The Graduate School asked each department to identify courses with a distance 
learning component comprising 25% or more of the total allocated contact hours. Instructors of 
each course identified were asked to sign a statement that the Quality Assurance Standards 
would be followed including student assessment of the course. Department chairs were also 
asked to provide their assurance that the standards would be met by signing at the bottom of the 
form. The courses for which there were signed forms became the universe of courses to be 
surveyed. 

(2) A code was entered into the survey database to identify each of these courses as 
distance learning (and therefore not subject to the normal SOIS). 

(3) An email message was sent to each instructor explaining the survey procedures and 
containing an embedded message for the instructors to forward to their students. The student 
message explained that we would send email messages to their ECU email account on November 
30th asking them to complete the survey and to provide a link to the survey website. It explained 
that they would need both their ECU email ID and their PIN to complete the survey. 



(4) As the November survey date approached, we noticed that only about one-third of the 
survey population had obtained PINs. Instructors were asked to emphasize the importance of 
accessing their ECU email accounts and obtaining PINs, in order to avail themselves of a rapidly 
increasing range of on-line student services, as well as to complete the survey. 

(5) The email messages were sent November 30 as planned. The messages included a 
link to the survey website. 

(6) When the student clicked on the link, they were sent to a survey entry screen where 
they were prompted to provide both their ECU email ID and their PIN. They were also given a 
link to obtain their PIN and as well as an opportunity to send a message to the survey 
administrator. 

(7) If the student's email ID and PIN could not be located in the database, or if they had 
already submitted a survey, they received a message "Unsuccessful Attempt to Access Survey 
Instrument" and were given an opporhmity to send a message to the survey administrator. 

(8) If the student's email ID and PIN were located and had not been used to submit a 
survey, the database was searched for distance education courses the student was enrolled in and 
those courses were displayed on a screen. 

(9) The student was asked to click on the course to be evaluated, click on the "Survey" 
button, and be taken to the survey instrument. A "Submit Survey Responses" button is located at 
the end of the instrument. 

(10) If one or more questions were left blank, a screen was displayed that identified the 
blank questions and asked the student to use the "back" button to return to the survey instrument 
to complete the unanswered questions. 



(1 1) If all questions were answered, a "Thank You" screen was displayed. The students 
were asked if they wanted to evaluate another distance learning course. If so, they were taken 
back to the sign-in page where they were prompted to re-enter their email ID and PIN for 
security purposes. 

IV. Initial Results and Conclusions 

When this paper was planned, this section was titled simply "Results and Conclusions." For 
reasons that will be apparent below, we changed the title to "Initial Results and Conclusions" and 
added sections titled "Modifications" and "Most Recent Results and Conclusions." 

Initial results CFall Semester 1998) : 

1. We achieved only a 20 percent response rate. This made it impossible to interpret the 
results as accurate reflections of student opinions for most of the classes siu^eyed. The low 
response was a surprise because we were working with graduate students who were computer 
literate — since most of them were taking their course via the Internet. All of the instructors had 
been notified that the survey would be conducted and had given their consent (by sigmng the 
Graduate School Quality Assurance Standards). The students had been notified of the survey 
procedures and had been urged to get their ECU PINs. 

We learned that the students taking graduate distance education courses are, for the most 
part, busy adults. One of the reasons that they chose distance education was convenience. These 
students do not have the time (or patience) to go through a series of procedures in order to 
complete a survey. They are willing to complete a siu^ey -and willing to provide extensive 
comment-only if it can be done quickly and easily. We also learned that many of the graduate 
distance education courses were self-paced and many of the students had completed the course 



well before the November 30th survey dates. We also had not considered that we were 
requesting their attention at one of the busiest times of the year - just before the Christmas 
holidays! 

2. The survey instrument itself worked beautifully! There was no evidence that the length 
of the survey was a problem. All of the students who entered the survey instrument answered all 
questions and most provided extensive comments. There was not a single negative comment 
about the survey instrument. 

Initial Conclusions: 

1 .We needed to develop more of a partnership with instructors since some of them 
perceived the assessment process as one imposed by the Graduate School and PIR. They did not 
feel as if they had been adequately consulted in the development of the instrument and 
procedures (even though many of them had been members of the committees that developed the 
questionnaire). 

2. We needed to improve student access to the survey instrument. The email ID and PIN 
requirements were too stringent since many of the distance education students did not have (and 
would not get) a PIN. Many of these students used their own internet provider with its distinct 
email addresses and did not understand that they had also been assigned an ECU email accoimt 
upon acceptance as a graduate student. 

3. Students were pleased with their distance learning courses. Table 1 shows a summary 
of the quality assurance questions from the surveys submitted. On a 7-point Likert scale, the 
average score was 6.0. The overall assessment was either a 6 or 7 for 86 percent of the cases. 
Over 95 percent of the respondents said they would take another distance education course. 
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4. Students were not as happy with their instructors in a distance education setting as they 
were in a classroom setting. (Perhaps they could not enjoy the instructors' magnetic personalities 
as much!) Several courses that were surveyed as distance learning were also taught by the same 
instructor in a classroom setting. The instructor received consistently higher scores from 
students in the classroom setting for all except two questions. The two exceptions were questions 
about the difficulty of the course- the students thought the distance education courses were more 
difficult. 

The lower scores for distance education may reflect the higher maturity and experience 
levels of distance education students. These students may be more discerning than their younger 
counterparts in classroom settings. 

Why did we get such a poor response rate? What went wrong? 

1. Poor knowledge of ECU email accounts . Few of the distance education students knew 
that they had been assigned an ECU email account. Even worse, few of the instructors knew that 
their students had been assigned an ECU email account. The instructors were all using student- 
provided home email accounts to conduct the course. 

2. Inadequate Comiminication with instructors. There was evidence that at least some 
instructors saw this as an extraneous activity that had been imposed on them. Most of the 
instructors had built into their on-line course their own evaluation surveys (and even then some 
had trouble getting good response rates). 

3. Awkward and difficult process to obtain a PIN. The students had to go through 
multiple steps to acquire-and then to activate-a PIN. (This process has since been streamlined.) 
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4. Adequate help facilities were not available to assist distance education students in the 



process of accessing their email accoxmts and obtaining PINs. 

5. Rigidity . The screen we had established to control access to the survey was 
unnecessarily rigid. It required that students recognize the difference between email ID and email 
address, that they type their email ID in upper case letters, and that they enter their PIN as an 8- 
digit field with leading zeros. 

6. Speed . After the student successfully entered their email ID and PIN, the system 
searched for and displayed the distance education coiirses that student was enrolled in. This 
process was, for a variety of reasons, often too slow. 

V. Modifications 

We restructured the survey process for the Spring 1999 semester to correct for the weaknesses 
discovered in the first (Fall 1 998) pilot survey. 

1 . Improved partnership with instructors. We held a meeting with instructors early in the 
Spring 1999 semester to discuss the survey and to seek their advice. The instructors lirged us to 
abandon the use of ECU email accoxmts and PINs for survey control. (Some instructors, 
however, had begim to require their students to access their ECU email accounts in order to 
participate in the coiirse.) Some instructors were even requiring their students to obtain ECU 
PINs. However, most of the instructors thought that their distance education students had no use 
for an ECU email accoimt, would never take the time to get a PIN, and would not participate in 
the survey if they could not do so easily without an ECU email ID or PIN. We also outlined a 
plan to conduct a test of the revised survey procediires using the instructors as the survey 
population. The test of the revised procediires will inform the instructors of the process we are 



asking their students to use, will seek further advice and comment from the instructors, and 
ultimately will help to ensure their full cooperation to obtain a good survey response from their 
students. 

2. Improved access to the survey instrument . We decided to use the student ID for survey 
control. (A 9-digit number required for registration- usually the student's SSN.) This will make 
the survey available to those distance education students who have not accessed their ECU email 
accoimt and/or have not obtained a PIN. We also made it possible to access the survey 
instrument from the student desktop. The student desktop web-page contains a variety of on-line 
student services that should be attractive to distance education students. 

3. Improved help for distance education students. We created a web page for distance 
education students to explain several ECU procedures including the distance education sxirvey. It 
provides information (and links wherever possible) to help on a variety of topics including email 
accoimts and PINs. 

4. Fine-tuning. Some further adjustments were made to improve the ease and speed of 
accessing the survey instrument. 

VI. Most Recent Results and Conclusions 

The Spring 1999 survey went much more smoothly and provided more useful results. 

1 . Response rate . The response rate improved to 43%— much better than the 20% 
response for the previous semester, but much worse than the 80-85% response from paper forms 
administered in the classroom. 

There was considerable variation in response rates by instructor, suggesting that the 
instructor's methods for encouraging the students to respond may be the single most important 
factor. The response rates by instructor ranged from 92% to 4%. We talked with instructors at 



both the high and the low ends of the response rates. The instructors with high response rates 
attributed their success to frequent communication with the students about the importance of 
student evaluations. In one case, the instructor built the survey into the course as the final step. 
One instructor made the survey part of the final exam. Confidentiality concerns, however, 
prevented us from providing the instructor with the names of students who had not responded, 
and there was less than a 100% response for this course. 

Instructors with low response rates pointed to institutional apathy on the part of their 
students. They pointed out that these are graduate students, usually busy adults, with a loose 
connection with ECU. Students of one course with an exceptionally low response rate were 
taking the course through a partnership with other schools and evaluated the course for the other 
school. Some instructors reported that students with older MAC computers had difficulty 
accessing the survey instrument. One instructor suggested that we offer alternative procedures 
whereby the student could download the survey form, fill it out, and mail it to us. 

2 . Student comments . The respondents took the time to provide extensive comments to 
the three questions at the end of the survey that requested written answers. This may turn out to 
be the most useful portion of the survey for instructors. 

3. Survey instrument . Once again, the survey instrument itself performed well. We 
received no complaints about the length of the survey and, as noted above, most respondents 
provided considerable written comment. There were some positive comments about the 
appearance and ease of use of the survey instrument. 

4 . Student satisfaction . Students continued to be pleased with distance education courses- 
- 93% of the respondents said they would take another distance education course. 
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The satisfaction ratings, however, were generally lower than those for the preceding 
semester. Overall, the average score for all questions (with a 7-point likert scale) asked of all 
respondents was 5.8, down from 6.0 the preceding semester. More importantly, the percentage 
of students responding 1 or 2 to the 7-point scale increased from 4.3% to 7.6%. The students 
rated their own preparation and the instructor 6.0 and 5.9, slightly higher than the preceding 
semester. They rated the course, the distance learning aspects of the course, and the overall 
assessment 5.7, 5.7, and 6.1, respectively-slightly lower than the preceding semester. However, 
none of the differences in ratings between semesters were significant. 

VII. Conclusions 

The most important conclusion that we want to share with our fellow institutional 
researchers is the difficulty of reaching distance education students with a web-based survey. 
This is somewhat coimter-intitutive because this is a student population that should be skilled in 
computer use, and in frequent one-on-one electronic contact with their instructors. However, 
these students tend to be older, busier, and less attached to the school than their classroom 
coimterparts. 

Secondly, we conclude that the survey development will be a much longer process than 
originally planned. We will need to continue talking with instructors and students and making 
improvements to the process. For example, we need to improve the process for identifying 
courses to be surveyed as distance education. Sometimes the courses reported to the Graduate 
School as distance education were instead taught in a classroom and more appropriately 
surveyed with traditional paper forms administered in the classroom. We also learned that 
instructors teaching distance education courses conducted for extension purposes through ECU 



Division of Continuing Studies confused those courses with regular graduate courses taught 
through their home departments and that process administrators shared in that confusion. This 
result indicated that internally ECU needs to more clearly distinguish which courses should be 
surveyed. 

Third, we conclude that this survey should be extended to undergraduate distance 
education courses. It is not clear that the problems encountered with graduate courses will be 
any worse (or better) with the undergraduate distance education courses that are presently not 
surveyed at all. 

Fourth, we conclude that we should continue to replace traditional paper surveys with 
web-based surveys with a good deal of caution and with deliberate slow speed. The benefits of 
web-based surveys are clear -- lower cost, more complete responses, and extensive student 
comments. The problems, however, are extensive staff time for the developmental phase of the 
survey, and lower response rates. 

Fifth, we also conclude that the political problems of developing web-based surveys for 
distance education courses will not be inconsiderable. Problems of securing approval from 
various campus constituencies for new student opinion of instruction instruments, concerns about 
response rates, concerns about the comparability of the results from traditionally taught courses 
and distance education, and the resulting uncertainty of impacts for faculty personnel decisions 
are major issues that ECU faces in the development of its web-based distance education surveys. 
We doubt we are alone. 
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